Appendix A is xe2x80x9cComputer Program Listing Appendixxe2x80x9d on a CD (Compact Disk). containing the following 17 text files consisting of 18427730 bytes.
Appendix B is a copy of forty two page co-pending application No. 60/188,524 filed Mar. 10, 2000.
Appendix C is a copy of twelve page co-pending application No. 60/188,553 filed Mar. 10, 2000.
Appendix D is a copy of a seven page document entitled xe2x80x9cUnification-based Multi-modal Parsingxe2x80x9d by Michael Johnson
Appendix E is a copy of a seven page document entitled xe2x80x9cCreating Tangible Interfaces by Augmenting Physical Objects with multi-modal Languagexe2x80x9d by David McGee and Philip Cohen.
Appendix F is a copy of a nine page document entitled xe2x80x9cSomething from Nothing: Augmenting a paper-based work practice via multimodal interactionxe2x80x9d by David R. McGee, Philip R. Cohen and Lizhong Wu.
All of the above listed appendices A to F are hereby incorporated herein in their entirety by reference.
The present invention relates to computer systems and more particularly to a method and system which provides for interaction between a user and a computer system.
The prior art provides a number of methods for computationally augmenting natural, non-digital settings. The term augmenting is herein used to mean extending a physical artifact so that it represents or denotes something more or something new, and thereby gains computation meaning and behavior.
A paper, by Underkoffler, J. and Ishii, H., Urp: xe2x80x9cA luminous-tangible Workbench for Urban Planning and Designxe2x80x9d, published in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ""99), May 1999, ACM
Press, 386-393 describes a system called xe2x80x9cUrpxe2x80x9d that planners use to build models, rulers, clocks, and other physical objects. Objects are tagged by patterns of colored dots, and if a pattern is recognized, a vision system sends Urp the associated object""s location. With Urp, augmented objects xe2x80x9cbehavexe2x80x9d as you would expect them to: rulers measure distances, clocks mark time, and so on. The object""s physical characteristics and the environment it inhabits govern these expectations
A paper by Wellner, P.,xe2x80x9cInteracting with paper on the DigitalDeskxe2x80x9d published in the Communications of the ACM, 1993. 36(7): 87-96 describes a system that augments office work by introducing paper into a workstation environment. Through computer vision, users can point at numbers on a real piece of paper, in response to which the system performs optical character recognition and pastes the recognized number into the system""s calculator. Similarly, regions of real paper, like a sketch on a napkin, can be cut and pasted into a painting program.
A paper by Ishii, H. and Ullmer, B., xe2x80x9cTangible bits: towards seamless interfaces between people, bits and atomsxe2x80x9d, published in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ""97), March 1997, ACM Press, 234-241 describes a system with a shared whiteboard, uses barcode-tagged cards to hold digital ink. However, the ink can only be retrieved when scanned by a barcode reader connected to a desktop computer.
A paper by Coen, M. H., xe2x80x9cDesign principles for intelligent environmentsxe2x80x9d published in the Proceedings of the Conference on Artificial Intelligence (AAAI ""98), July 1998, American Association for Artificial Intelligence, 547-554 describes a system that uses Post-it Notes to activate behaviors in the room. Different colored Post-it Notes are used so that they can be easily distinguished from each other and from the supporting table by a vision system. Ink on each note is used only to remind the user of the action to expect, not as input to the system.
A system call xe2x80x9cPassage xe2x80x9d is described in a paper by Streitz, N. A., Geibler, J., and Holmer, T., xe2x80x9cRoomware for cooperative buildings: integrated design of architectural spaces and information spacesxe2x80x9d published in the Proceedings of the First International Workshop on Cooperative Buildings: Integrating Information, Organization, and Architecture (CoBuild ""98), February 1998, Springer-Verlag, 4-21 and a technique termed RFID (radio frequency identifier) is described in a paper by Want, R., Fishkin, K. P., Gujar, A., and Harrison, B. L., Bridging physical and virtual worlds with electronic tags, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ""99), May 1999, ACM Press, 370-377. These systems interpret the human act of augmenting paper in order to create a digital representation and provide some flexibility in changing that data. Within the Passage concept, meaning can be linked graphically to a physical object whenever that object is placed on a xe2x80x9cbridge.xe2x80x9d In the initial prototype, the bridge is a scale and recognizes objects based on their weight. With the RFID system, tags are hidden in books, documents, watches, etc. As with Passage, associational augmentations can be formed when the tags are first detected. These systems do not yet support a pre-existing work practice, nor can users learn what information is associated with an object unless the users and the object are adjacent to a bridge or detector. More generally, associational augmentation methods like these and others, such as the use of colored dots, glyphs, or bar codes, fail to present the linked digital information to the user without the assistance of technology.
A paper by Moran, T. P., Saund, E., Melle, W. V., Bryll, R., Gujar, A. U., Fishkin, K. P., and Harrison, B. L., xe2x80x9cThe ins and outs of collaborative walls: Demonstrating the Collaborage conceptxe2x80x9d, published in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ""99), May 15-20, 1999, ACM Press, CHI""99 Extended Abstracts, 192-193T herein called the Collaborage concept. The Collaborage concept characterizes augmented systems consisting of a board and various tagged physical information items. Several such prototypes have been built. One of these prototypes is an In/Out board system with glyph-tagged magnetized photos that can be slid from the Out column to the In column and vice-versa. Within seconds, a vision system recognizes the change in location of the glyph and an In/Out web page is updated to reflect the change in status. If the system were to fail, individuals could still check the physical In/Out board, move their picture from one column to the other, add hand-written annotations, and walk away with up-to-date information. Because objects are augmented using glyphs rather than a natural language, users cannot easily add new digital information to the board. For example, a new employee cannot use any magnet and photograph and expect it to work in the Collaborage system.
The present invention provides a method and system designed to augment, rather than replace, the work habits of its users. These work habits include practices such as drawing on secondary physical documents (e.g. Post-it(trademark) notes) using a symbolic language and placing these secondary physical documents on a primary physical document such as a map. With the present invention the system observes and understands the language used on the primary and secondary physical documents and the system assigns meaning simultaneously to objects in both the physical and virtual worlds. Since the data is preserved in physical form, the physical documents serve as a back-up in the case of electronic system failure. With the present invention users can utilize a paper primary physical document such as a map, register it, and augment the primary physical document by placing secondary physical documents on the primary physical document. The secondary physical documents can be moved from one place to another on the primary document. Once an object is augmented, users can modify the meaning represented by it, ask questions about that representation, view it in virtual reality, or give directions to it, all with speech and gestures.